Handover procedure for driver of autonomous vehicle

ABSTRACT

A control system that is operable to control a vehicle in an autonomous or semi-autonomous mode includes a processor that processes data captured by a plurality of exterior sensing sensors. When the control system is operating in the autonomous or semi-autonomous mode and responsive to a determination of an upcoming event that requires the system to hand over control of the vehicle to a driver before the vehicle encounters the event, the control determines (i) a total action time until the vehicle encounters the event, (ii) an estimated time for the driver to take over control and (iii) an estimated handling time for the vehicle to be controlled before the vehicle encounters the event. Responsive to the determinations, the control system (i) allows the driver to take over control of the vehicle or (ii) controls the vehicle to slow down and stop the vehicle before the vehicle encounters the event.

CROSS REFERENCE TO RELATED APPLICATION

The present application claims the filing benefits of U.S. provisionalapplication Ser. No. 62/401,310, filed Sep. 29, 2016, which is herebyincorporated herein by reference in its entirety.

FIELD OF THE INVENTION

The present invention relates generally to a vehicle vision system for avehicle and, more particularly, to a vehicle vision system that utilizesone or more cameras at a vehicle.

BACKGROUND OF THE INVENTION

Use of imaging sensors in vehicle imaging systems is common and known.Examples of such known systems are described in U.S. Pat. Nos.5,949,331; 5,670,935 and/or 5,550,677, which are hereby incorporatedherein by reference in their entireties.

SUMMARY OF THE INVENTION

The present invention provides a driver assistance system or controlsystem for a vehicle that provides autonomous or semi-autonomous controlof the vehicle, such as responsive to image processing of image datacaptured by the cameras and processing of sensor data captured by othersensors of the vehicle, such as radar sensors or the like. The system isoperable to determine when an occupant or driver of the vehicle shouldtake over control of the vehicle (from the autonomous or semi-autonomouscontrol mode) (such as responsive to a determination of a hazardousdriving condition or a determination of a fault or defect or error withthe autonomous or semi-autonomous control of the vehicle) and determinesor estimates how long it will take the driver to take over control ofthe vehicle and, responsive to the determinations and to the type ofsituation requiring the driver to take over, the system may continue tocontrol the vehicle to avoid a hazard or to slow down or stop thevehicle until the driver is ready to take over control of the vehicle.The system continues to control the vehicle or releases control to thedriver depending on (i) how much time the system determines is availablefor the driver to safely take over control of the vehicle and to steeror maneuver the vehicle to avoid a hazard or the like, (ii) how muchtime the system estimates it will take the driver to take over controlof the vehicle, and (iii) the maximum takeover time in which the drivercan take over control of the vehicle sufficiently prior to the eventpoint.

For example, if the estimated recovery time of the driver is greaterthan the takeover time, then the system continues to control the vehicleand executes a safe harbor maneuver. If the estimated recovery time ofthe driver is less than the takeover time, then the system allows thedriver to take over control of the vehicle. If the estimated recoverytime of the driver is less than the takeover time, but the driver doesnot take over in time, then the system continues to control the vehicleand executes a safe harbor maneuver.

These and other objects, advantages, purposes and features of thepresent invention will become apparent upon review of the followingspecification in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a plan view of a vehicle with a vision system thatincorporates cameras in accordance with the present invention;

FIG. 2 is a time scheme at which an estimated driver recovery timeexceeds the evaluated available time to hand the vehicle control back tothe driver (Hand Over Time), such that the vehicle triggers a SafeHarbor Maneuver right from the beginning and not exceeding the PlannedHandling Time;

FIG. 3 is a time scheme at which the estimated driver recovery time isless than the evaluated available time to hand the vehicle control backto the driver (Hand Over Time), such that the vehicle does not trigger aSafe Harbor Maneuver (SHM), and because the point of time when thedriver actually takes over is earlier than the Latest Hand Over point intime, the system allows the driver to take over control of the vehicle;

FIG. 4 is a time scheme at which the estimated driver recovery time isless than the evaluated available time to hand the vehicle control backto the driver (Hand Over Time), but since the point of time when thedriver really is ready to take over is later than the Latest Hand Over(LHO) point in time, the vehicle initiates a SHM at the LHO, and the SHMtakes as long as the Planned Handling Time; and

FIGS. 5, 6 and 7 are the same time schemes as FIGS. 2, 3 and 4,respectively, with the exception that the system's redundancy has beenlost since a specific required autonomous driving system component ordevice is defective or has failed.

DEFINITIONS

Action Time—Remaining time before reaching an event that either triggersHDAR or HDIR.

Human Driver Attention Request (HDAR)—The system requests attention fromthe driver to observe the current driving situation.

Human Driver Intervention Request (HDIR)—The system requests the driverto take over the driving task while in an active state or transitionstate (this may imply a limited access highway or LAH).

Driver Handling Time—The time that the driver needs to perform necessarymaneuver.

Planned Handling Time—The time the system expects a maneuver to takewhen executed, where this is the larger value of either the driverhandling time or the time for a safe harbor maneuver (where it may beunclear whether the driver will take over).

Estimated Recovery Time—Estimated time it takes the driver to take overcontrol of the subject vehicle, which may be determined from drivermonitoring, and estimates how long it will take until the driver ismentally and physically ready to perform the next maneuver.

Event Point—An event that cannot be handled by the system, such as, forexample, when at x seconds from a given time (or a distance ahead of thevehicle), the vehicle will have a catastrophe if nothing is done (e.g.,encounter an object or experience a system failure).

Handover Time—Theoretically available time for handover (the differencebetween the action time and the planned handling time or safe harbormaneuver time).

Redundancy Undershoot Condition (“Red-Condition”)—Functional safetyconcept is violated due to loss off redundancy caused by technicalfailure.

Safe Harbor Maneuver Time—The time necessary to execute a maneuverseeking safe harbor.

Driver Take Over Time (at first named Driver Reaction Time)—The time itreally takes the driver to take control of the subject vehicle.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

A driver assist system and/or object detection system and/or alertsystem and/or autonomous vehicle control system operates to captureimage data and/or sensor data exterior of the vehicle and may processthe captured image data to display images and to detect objects at ornear the vehicle and in the predicted path of the vehicle, such as tocontrol the vehicle in an autonomous or semi-autonomous mode. The systemincludes a processor that is operable to receive data from one or morecameras and/or other sensors (such as radar sensors or the like) and maydetect the path of travel of the vehicle and/or determine the presenceof objects in the path of travel of the vehicle and/or may provide anoutput to a display device for displaying images representative ofcaptured image data. Optionally, the vision system may provide display,such as a rearview display or a top down or bird's eye or surround viewdisplay or the like.

Referring now to the drawings and the illustrative embodiments depictedtherein, a vehicle 10 includes a sensing system 12 that includes atleast one exterior facing sensor (such as a camera and/or radar sensorand/or the like), such as a rearward facing imaging sensor or camera 14a (and the system may optionally include multiple exterior facingimaging sensors or cameras and/or other sensors, such as a forwardfacing camera 14 b (and/or other sensors) at the front (or at thewindshield) of the vehicle, and a sideward/rearward facing camera 14 c,14 d (and/or other sensors) at respective sides of the vehicle), whichcaptures image data exterior of the vehicle, with the camera having alens for focusing images at or onto an imaging array or imaging plane orimager of the camera (FIG. 1). Optionally, a forward viewing camera maybe disposed at the windshield of the vehicle and view through thewindshield and forward of the vehicle, such as for a machine visionsystem (such as for traffic sign recognition, headlamp control,pedestrian detection, collision avoidance, lane marker detection and/orthe like). The system 12 includes a control or electronic control unit(ECU) or processor 18 that is operable to process data captured by thecamera or cameras or sensors and may detect objects or the like and/orprovide displayed images at a display device 16 for viewing by thedriver of the vehicle (although shown in FIG. 1 as being part of orincorporated in or at an interior rearview mirror assembly 20 of thevehicle, the control and/or the display device may be disposed elsewhereat or in the vehicle). The data transfer or signal communication fromthe camera to the ECU may comprise any suitable data or communicationlink, such as a vehicle network bus or the like of the equipped vehicle.

According to SAE Level 3, autonomous vehicles still are not required tohandle all driving and emergency tasks which may come up during drivingby their own, and due to that, it is still required that the driver maytake back the vehicle control. In known commercially availablesolutions, such as, for example, the Daimler E-Class and Tesla Svehicles, the vehicle is typically requesting the driver to take backcontrol by first actuating the taking over alarms, such as beeping,displaying and vibrating, and then falling into a state at which thevehicle continuously decelerates in case the driver does not take over.

Although the examples are not intended to fulfill Level 3, they complymore or less with SAE level 2. Both systems require that the driver isattentive at all times, every time capable to take back control from theautomated driving system. Both of these systems are meant forhighway-like (essentially intersection free) roads only (limited accesshighway (LAH) or closed access highway (CAH)) and both are not capableof guiding the vehicle off the drive lanes as system emergencyperformance. In an emergency driving condition, there is no redundancyfor the driver himself or herself. Systems according to SAE Level 3instead require that the system is capable to handle dynamic drivingtasks but a driver is present as redundancy.

Some commercial systems are nowadays equipped with an e-call system suchas Jaguar Land Rover's ‘Land Rover Incontrol™’. These systems are ableto detect a state of emergency in certain conditions. Especially crashsituations get detected (especially the air bag deployment) at which anautomated call to an emergency service is triggered by the vehicleautomatically. Multi collision braking (MCB) systems such asVolkswagen's ‘Volkswagen Multi Collision Brake’ are known to brake pasta collision detection to mitigate or avoid subsequent collisions. Thisputs the vehicle in a potentially safer state than to rely on thedriver's potential braking interaction, since the driver may be unableto brake. The driver's condition stays typically undetected by thesekind of systems. An e-call system with driver health condition detectionby processing vehicle inherent and non-vehicle inherent sensor's data isdescribed in U.S. patent application Ser. No. 15/463,293, filed Mar. 20,2017, and published on Sep. 28, 2017 as U.S. Patent Publication No.US-2017-0274906, which is hereby incorporated herein by reference in itsentirety.

For autonomous vehicles suitable for deployment with the system of thepresent invention, an occupant of the vehicle may, under particularcircumstances, be desired or required to take over operation/control ofthe vehicle and drive the vehicle so as to avoid potential hazard for aslong as the autonomous system relinquishes such control or driving. Suchoccupant of the vehicle thus becomes the driver of the autonomousvehicle. As used herein, the term “driver” refers to such an occupant,even when that occupant is not actually driving the vehicle, but issituated in the vehicle so as to be able to take over control andfunction as the driver of the vehicle when the vehicle control systemhands over control to the occupant or driver or when the vehicle controlsystem is not operating in an autonomous or semi-autonomous mode.

Typically an autonomous vehicle would be equipped with a suite ofsensors, including multiple machine vision cameras deployed at thefront, sides and rear of the vehicle, multiple radar sensors deployed atthe front, sides and rear of the vehicle, and/or multiple lidar sensorsdeployed at the front, sides and rear of the vehicle. Typically, such anautonomous vehicle will also have wireless two way communication withother vehicles or infrastructure, such as via a car2car (V2V) or car2xcommunication system.

The present invention provides a system that controls the takeoverprocess where the driver takes over control of an autonomous orsemi-autonomous vehicle. For giving the driver the freedom to not beingattentive to the road scene while driving in autonomous mode (SAE Level3 mode), it is the usual approach to hand over the vehicle before adriving task emerges that only a human can handle (commonly referred toas a Human Driver Intervention Request HDIR). The driver has to takeover before the driving task emerges unless the system redundancy islost. Such a task is often referred to as a driver intervention task(DIT). The driver is also the ‘fall back’ redundancy when an automateddriving system function is failing as discussed below. In practice thereis the matter that a situation which must be handled by a human may comefaster than it takes a human driver to take over (driver recovery DR),where this may be because the driver is distracted or sleeping, orbecause the HDIR came very fast that the system hand over and thereaction time of a human driver in sum would take too long.

Previously known in automotive is to have driver assistant systems thatdetect the environmental scene, especially the scene of the path aheadthe subject vehicle (SV) or ego vehicle is planning to take. In case ofcollision hazard events (CHE), such as, for example, when a piece oflost cargo (hazardous object, HO) appears at the lane of the subjectvehicle (SV), the systems are able to calculate the time to collision(TTC) when taking the speed (and the HO's static or changing speed isconstant) into account and an extended time to collision in case thevehicle assistant system is taking collision mitigating actions such asbraking into account. Some systems may be able to also execute collisionavoiding steering maneuvers in combination with braking, but thisrequires a proper scene context understanding, so that the system doesnot make worse the CHS situation rather than do any good (since thescene detection is not perfect there may be objects such as pedestrianswhich may have been overlooked by the system and thus may be harmedduring the collision mitigation maneuver, while the vehicle is avoidingthe HO instead). Because of that, the emergency handling is still leftto the human drivers (with the exception of MCB brake actuation past acollision detection).

In accordance with an aspect of the system of the present invention, avehicle with automated driving feature SAE Level 3 may have a subsystemto detect the driver's ability to take over, and is capable to do anestimation of the time it will take the driver to take over control ofthe vehicle (estimated driver recovery time EDRT), see FIGS. 2, 3 and 4.The system may have in-cabin sensors and optionally the sensinginformation of smartphones and optionally the sensors of wearables orimplants of the driver to monitor whether the driver is attentivelydriving or which non driving related activities he or she is doing orhow far he or she is distracted and optionally which health conditionsystem he or she is in. The estimated hand over time may be picked outof a look up table of previous mass screening results of the typicaltime the specific non driving activity is taking. Upon the underrun ofone or more specific or combined health or distraction parameters, thesystem may assume the driver is unable to take over (or recover) thevehicle (see case in FIG. 2).

Optionally, there may be health parameters indicating the driver issleeping. Optionally, the system may have the ability to learn thedriver's specific health parameters in a calibration phase or thatparameters may be given from elsewhere as specific teaching data set.Optionally, the estimation of the hand over time may depend partially onthe driver's age. Optionally, the estimation of the hand over time maydepend partially on the driver's sex. Optionally, the estimation of thehand over time may depend partially on the driver's ethnos. Optionally,the estimation of the hand over time may depend partially on how longthe driver was already awake continuously. Optionally, the estimation ofthe hand over time may depend partially on whether the driver has hadthe possibility to have a nap while he or she was awake. Optionally, theestimation of the hand over time may depend partially on whether thedriver has eaten lately. Optionally, the estimation of the hand overtime may depend partially on the time of day or night. Optionally, theestimation of the hand over time may depend partially on the date orseason or driving/environment conditions. Optionally, the estimation ofthe hand over time may depend partially on the in cabin climateparameters (AC is on or off, it's cold outside or sunny). Optionally,the estimation of the hand over time may depend partially on the lightconditions. Optionally, the estimation of the hand over time may dependpartially on the work load or stress [level] the driver may have had inthe recent past before or while driving.

Optionally, the estimation of the hand over time may depend partially onthe facts whether the driver is driving alone or driving with occupantsand whether he or she is in conversation with them. Optionally, theestimation of the hand over time may depend partially on whether thedriver is on the phone. Optionally, the estimation of the hand over timemay depend partially on whether the driver is texting. Optionally, theestimation of the hand over time may depend partially on whether thedriver is eating or drinking. Optionally, the estimation of the handover time may depend partially on whether the driver is reading.Optionally, other factors may also or otherwise be taken into accountfor the estimation of the hand over time in case these are detectableand classifiable, such as, for example, when the driver is paintingfingernails, applying makeup or shaving or the like, when the driver isnose picking, when the driver has climbed to the back seat, when thedriver leans out of the window or sunroof, when the driver is smoking,when the driver has covered his or her eyes with anything.

Optionally, instead of using a look up table, there may be an artificialintelligence (AI) algorithm, such as, for example, a reward feedbacklearning type AI, implemented in the system that is substantiallycontinuously watching the detectable or accessible driver health andactivity parameters that are suitable to assess whether the driver isable to take over (recover). Especially when the driver is using thevehicle frequently, the system may continuously improve the assessmentof the driver's health, activity and distraction parameters and theassociated time it predicts the driver needs to take over control of thevehicle and the time it really takes the driver to take over control ofthe vehicle. Optionally, the AI algorithm may have the ability to alsoassimilate the learning input of other equipped vehicle fleet membersvia V2V or V2X for continuously improving the driver recovery timeestimation.

As an additional inventive aspect of the system of the presentinvention, the system may have according environmental sensors and anenvironmental scene detection processing algorithms as referred above.Optionally, the system may have a v2x and/or v2v connection forreceiving additional scene map and hazardous object (HO) data and datato calibrate the SV ego position. Once the HO is known, the system ofthe invention may be able to predict evasive paths and necessary brakingmaneuvers, may be able to predict the time the maneuver will take andmay assess whether it is necessary to put the control back to the driveras redundant handler for dynamic situations. Because the systemcontinuously assesses or estimates the driver recovery time (EDRT), thesystem can calculate the threshold whether the EDRT is shorter or longerthan the shortest time interval (Planned Handling Time or PHT) that ispredicted that a human driver would need to properly control the vehiclein a safe manner in face of an upcoming event. The upcoming event may beto stop (see FIGS. 5, 6, and 7) in time before hitting a hazardousobject or may be something comparably harmless such as for the driver torecover vehicle control when taking a highway off ramp. The remaininghand over time (HOT) is the result of subtracting the PHT from the totalremaining time (Action time). The Driver Take Over Time (DTOT) is thetime it really takes until the driver takes over.

In accordance with the system of the present invention, in case theestimated driver recovery time or EDRT is longer than the hand over timeor HOT (see case shown in FIG. 2), the vehicle will initiate a safeharbor maneuver SHM immediately, since there will be a timeframe atwhich no driver redundancy is given (predictively). Since redundancy isrequired, the SHM is the safest remaining option.

A SHM is any maneuver which brings the vehicle to the relatively safeststopping point. The path may be drafted by the above discussedenvironmental scene detection processing algorithms and paths planners.The path plan may involve lane changes. On right hand traffic highwaysor Autobahns, the planner may substantially plan to change subsequentlyto the righter lanes and then change to the shoulder or break down laneto stop. Optionally, it may be assessed that the safest stopping pointis a highway or Autobahn parking lot (‘Autobahnparkplatz’) in neardistance. In that case the system may change to the right lane untilreaching the lane leading to the parking lot and may park at the parkinglot (optionally utilizing an automated parking system's procedure).Optionally, the system may assess or determine that the safest stoppingpoint is a parking bay in near distance. In that case the system maychange to the right lane until reaching the parking bay and may stopthere.

In all cases where the EDRT is shorter than the HOT, the vehicle willallow the driver to take back drive control prior to the plannedhandling maneuver becoming necessary (at the end of the (real) drivertake over time or DTOT) (see case shown in FIG. 3). In cases where itturns out the driver did not take over the control within the ERT (seecase shown in FIG. 4), whether because the recovery time estimation waswrong or the driver just refused to take over in time, the vehicleinitiates a safe harbor maneuver or SHM (at the latest handover time(LHO). Depending on the type of event (Event on the Event Point—EventPoint is the point of time at which the event occurs or when or wherethe vehicle will encounter the event, such as a system failure or thecontrol's loss of ability to control the vehicle in the autonomous orsemi-autonomous mode or a hazardous condition or object ahead of thevehicle and in the path of travel of the vehicle), the driver may or maynot be allowed to take back the vehicle control after when the SHM wasalready triggered and partially executed. Some events demand severemaneuvers which may be adversely effected or messed up when a humandriver interferes since he or she does not fully understand it in timeor is not able to execute the maneuver. In some cases the vehicle mayexecute vehicle controls that are typically not an option to a humandriver, such as putting different torque to different wheels such asvehicle control systems with environmental scene detection and invasivepath planning may do. Such systems are described in U.S. Publication No.US-2017-0144658, which is hereby incorporated herein by reference in itsentirety.

As discussed above, the driver is required as being the ‘fall back’redundancy when an autonomous driving system component is failing. Inaccordance with another aspect of the invention, the system may requestthe driver to take back control (recover) via HDIR in case an autonomousdriving system component is detected as failing. Failing of an essentialautonomous driving system component means the required redundancy isabruptly lost (see cases shown in FIGS. 5, 6 and 7). There may be urgentcases at which it is tasked to prevent an emergency situation, such as,for example, the scene detection may be limited due to, for example, adefect of the front RADAR. In these cases, the system may request tohand over the driving task to the driver as soon as possible, optionallyinitiating a safe harbor maneuver or SHM simultaneously (see case shownin FIG. 5). On the other hand there may be less urgent cases (see casesshown in FIGS. 6 and 7) of autonomous driving system component failingsor faults, such as, for example, a defective front head light (Xenonlight bulb, or LED or the like) during day time and bright whethercondition. In these less urgent cases, the vehicle may inform the driverabout the defect and may trigger a HDIR but may not trigger a SHMimmediately but may reflect the estimated DRT. As a first option thesystem may stay in the automated driving mode, ignoring the less severedefect of an autonomous driving system component, such as upon aquitting of the defect message by the driver. Nevertheless, in theseless urgent cases the vehicle may be able to continue the automateddriving task for a long time without trouble, and the system may handover the driving task to the human anyway since it cannot predict whereor when the defective system component may become essential to continuedriving. While driving in, for example, a tunnel entrance is very wellpredictable, other conditions, such as, for example, a heavy rain showersetting in, is not very well predictable. As a second option, past theDRT the system may engage an SHM automatically, which may beinterruptible by the driver taking over control of the vehicle.Optionally, as soon the driver has taken over driving control once, thesystem may refuse to get (re-)engaged into automated driving modeaccording SAE Level 3 again, since the redundancy criteria is not givenwhen an autonomous driving system component is defective or failing.

Optionally, once an autonomous driving system component is determined tobe defective or failed, the system may provide driver attendedautonomous driving mode in accordance with SAE Level 2 as a morecomfortable mode than the driver just driving by himself or herself orin a mode according SAE Level 1. Optionally, when an autonomous drivingsystem component is determined to be defective or failed, the system mayprovide a lane and/or ACC assist mode in accordance with SAE Level 1 asa more comfortable mode than the driver just driving by himself/herself.Optionally, both modes may be offered to the driver to engage inaccordance with the remaining functioning sensors, devices andprocessing systems, staying on the safe side during providing thehighest comfort always.

Thus, the present invention provides a control system for a vehicle thatis operable to control the vehicle in an autonomous or semi-autonomousmode, and that is operable to either stop the vehicle or hand overcontrol of the vehicle to an occupant (which then becomes the driver) ofthe vehicle when the system determines that the vehicle is approachingan event or event point at which the control should not be operating inthe autonomous or semi-autonomous mode. The event may comprise ahazardous driving condition (such as a hazardous road condition or suchas an object or obstacle) ahead of the vehicle or an imminent failure insaid control system that may affect said control system's ability tocontrol the vehicle in the autonomous or semi-autonomous mode.

When the control system is operating in the autonomous orsemi-autonomous mode and responsive to a determination of the upcomingevent that requires the system to hand over control of the vehicle to adriver of the vehicle before the vehicle encounters the event, thecontrol determines (i) a total action time until the vehicle encountersthe event, (ii) an estimated time for the driver to take over control ofthe vehicle and (iii) an estimated handling time for the vehicle to becontrolled before the vehicle encounters the event. Responsive to thedetermined total action time, estimated driver take over time andestimated handling time, the control system either (i) allows the driverto take over control of the vehicle or (ii) controls the vehicle to slowdown and stop the vehicle before the vehicle encounters the event.Responsive to the estimated driver takeover time being greater than thedifference between the total action time and the estimated handlingtime, the control system controls the vehicle to slow down and stop thevehicle. Responsive to the estimated driver takeover time being lessthan the difference between the total action time and the estimatedhandling time, the control system allows the driver to take over controlof the vehicle, and responsive to the control allowing the driver totake over control of the vehicle but when the driver fails to take overcontrol of the vehicle by the difference between the total action timeand the estimated handling time, the control system controls the vehicleto slow down and stop the vehicle.

The vehicle may include any type of sensor or sensors, such as imagingsensors or radar sensors or lidar sensors or ladar sensors or ultrasonicsensors or the like. The cameras or sensors at the vehicle may compriseany suitable camera or sensor. The system includes an image processoroperable to process image data captured by the camera or cameras, suchas for detecting objects or other vehicles or pedestrians or the like inthe field of view of one or more of the cameras. For example, the imageprocessor may comprise an image processing chip selected from the EyeQfamily of image processing chips available from Mobileye VisionTechnologies Ltd. of Jerusalem, Israel, and may include object detectionsoftware (such as the types described in U.S. Pat. Nos. 7,855,755;7,720,580 and/or 7,038,577, which are hereby incorporated herein byreference in their entireties), and may analyze image data to detectvehicles and/or other objects. Responsive to such image processing, andwhen an object or other vehicle is detected, the system may generate analert to the driver of the vehicle and/or may generate an overlay at thedisplayed image to highlight or enhance display of the detected objector vehicle, in order to enhance the driver's awareness of the detectedobject or vehicle or hazardous condition during a driving maneuver ofthe equipped vehicle.

For example, the vision system and/or processing and/or camera and/orcircuitry may utilize aspects described in U.S. Pat. Nos. 9,233,641;9,146,898; 9,174,574; 9,090,234; 9,077,098; 8,818,042; 8,886,401;9,077,962; 9,068,390; 9,140,789; 9,092,986; 9,205,776; 8,917,169;8,694,224; 7,005,974; 5,760,962; 5,877,897; 5,796,094; 5,949,331;6,222,447; 6,302,545; 6,396,397; 6,498,620; 6,523,964; 6,611,202;6,201,642; 6,690,268; 6,717,610; 6,757,109; 6,802,617; 6,806,452;6,822,563; 6,891,563; 6,946,978; 7,859,565; 5,550,677; 5,670,935;6,636,258; 7,145,519; 7,161,616; 7,230,640; 7,248,283; 7,295,229;7,301,466; 7,592,928; 7,881,496; 7,720,580; 7,038,577; 6,882,287;5,929,786 and/or 5,786,772, and/or U.S. Publication Nos.US-2014-0340510; US-2014-0313339; US-2014-0347486; US-2014-0320658;US-2014-0336876; US-2014-0307095; US-2014-0327774; US-2014-0327772;US-2014-0320636; US-2014-0293057; US-2014-0309884; US-2014-0226012;US-2014-0293042; US-2014-0218535; US-2014-0218535; US-2014-0247354;US-2014-0247355; US-2014-0247352; US-2014-0232869; US-2014-0211009;US-2014-0160276; US-2014-0168437; US-2014-0168415; US-2014-0160291;US-2014-0152825; US-2014-0139676; US-2014-0138140; US-2014-0104426;US-2014-0098229; US-2014-0085472; US-2014-0067206; US-2014-0049646;US-2014-0052340; US-2014-0025240; US-2014-0028852; US-2014-005907;US-2013-0314503; US-2013-0298866; US-2013-0222593; US-2013-0300869;US-2013-0278769; US-2013-0258077; US-2013-0258077; US-2013-0242099;US-2013-0215271; US-2013-0141578 and/or US-2013-0002873, which are allhereby incorporated herein by reference in their entireties. The systemmay communicate with other communication systems via any suitable means,such as by utilizing aspects of the systems described in InternationalPublication Nos. WO 2010/144900; WO 2013/043661 and/or WO 2013/081985,and/or U.S. Pat. No. 9,126,525, which are hereby incorporated herein byreference in their entireties.

The system may also communicate with other systems, such as via avehicle-to-vehicle communication system or a vehicle-to-infrastructurecommunication system or the like. Such car2car or vehicle to vehicle(V2V) and vehicle-to-infrastructure (car2X or V2X or V2I or 4G or 5G)technology provides for communication between vehicles and/orinfrastructure based on information provided by one or more vehiclesand/or information provided by a remote server or the like. Such vehiclecommunication systems may utilize aspects of the systems described inU.S. Pat. Nos. 6,690,268; 6,693,517 and/or 7,580,795, and/or U.S.Publication Nos. US-2014-0375476; US-2014-0218529; US-2013-0222592;US-2012-0218412; US-2012-0062743; US-2015-0251599; US-2015-0158499;US-2015-0124096; US-2015-0352953; US-2016-0036917 and/orUS-2016-0210853, which are hereby incorporated herein by reference intheir entireties.

The system may utilize aspects of head and face direction and positiontracking systems and/or eye tracking systems and/or gesture recognitionsystems (such as to determine the driver attentiveness or the like).Such head and face direction and/or position tracking systems and/or eyetracking systems and/or gesture recognition systems may utilize aspectsof the systems described in U.S. Publication Nos. US-2016-0137126;US-2015-0352953; US-2015-0296135; US-2015-0294169; US-2015-0232030;US-2015-0022664; US-2015-0015710; US-2015-0009010 and/orUS-2014-0336878, which are hereby incorporated herein by reference intheir entireties.

The system may utilize sensors, such as radar or lidar sensors or thelike. The sensing system may utilize aspects of the systems described inU.S. Pat. Nos. 9,753,121; 9,689,967; 9,599,702; 9,575,160; 9,146,898;9,036,026; 8,027,029; 8,013,780; 6,825,455; 7,053,357; 7,408,627;7,405,812; 7,379,163; 7,379,100; 7,375,803; 7,352,454; 7,340,077;7,321,111; 7,310,431; 7,283,213; 7,212,663; 7,203,356; 7,176,438;7,157,685; 6,919,549; 6,906,793; 6,876,775; 6,710,770; 6,690,354;6,678,039; 6,674,895 and/or 6,587,186, and/or International PublicationNo. WO 2011/090484 and/or U.S. Publication Nos. US-2017-0254873;US-2017-0222311 and/or US-2010-0245066, and/or U.S. patent applicationSer. No. 15/675,919, filed Aug. 14, 2017, and published on Feb. 15, 2018as U.S. Patent Publication No. US-2018-0045812, Ser. No. 15/647,339,filed Jul. 12, 2017, now U.S. Pat. No. 10,239,446, Ser. No. 15/619,627,filed Jun. 12, 2017, and published on Dec. 14, 2017 as U.S. PatentPublication No. US-2017-0356994, Ser. No. 15/584,265, filed May 2, 2017,and published on Nov. 2, 2017 as U.S. Patent Publication No.US-2017-0315231, and/or Ser. No. 15/467,247, filed Mar. 23, 2017, andpublished on Sep. 28, 2017 as U.S. Patent Publication No.US-2017-0276788, and/or International PCT Application No.PCT/IB2017/054120, filed Jul. 7, 2017, and published on Jan. 11, 2018 asInternational PCT Publication No. WO 2018/007995, which are herebyincorporated herein by reference in their entireties.

Optionally, the vision system may include a display for displayingimages captured by one or more of the imaging sensors for viewing by thedriver of the vehicle while the driver is normally operating thevehicle. Optionally, for example, the vision system may include a videodisplay device, such as by utilizing aspects of the video displaysystems described in U.S. Pat. Nos. 5,530,240; 6,329,925; 7,855,755;7,626,749; 7,581,859; 7,446,650; 7,338,177; 7,274,501; 7,255,451;7,195,381; 7,184,190; 5,668,663; 5,724,187; 6,690,268; 7,370,983;7,329,013; 7,308,341; 7,289,037; 7,249,860; 7,004,593; 4,546,551;5,699,044; 4,953,305; 5,576,687; 5,632,092; 5,677,851; 5,708,410;5,737,226; 5,802,727; 5,878,370; 6,087,953; 6,173,508; 6,222,460;6,513,252 and/or 6,642,851, and/or U.S. Publication Nos.US-2012-0162427; US-2006-0050018 and/or US-2006-0061008, which are allhereby incorporated herein by reference in their entireties. Optionally,the vision system (utilizing the forward facing camera and a rearwardfacing camera and other cameras disposed at the vehicle with exteriorfields of view) may be part of or may provide a display of a top-downview or birds-eye view system of the vehicle or a surround view at thevehicle, such as by utilizing aspects of the vision systems described inInternational Publication Nos. WO 2010/099416; WO 2011/028686; WO2012/075250; WO 2013/019795; WO 2012/075250; WO 2012/145822; WO2013/081985; WO 2013/086249 and/or WO 2013/109869, and/or U.S.Publication No. US-2012-0162427, which are hereby incorporated herein byreference in their entireties.

Changes and modifications in the specifically described embodiments canbe carried out without departing from the principles of the invention,which is intended to be limited only by the scope of the appendedclaims, as interpreted according to the principles of patent lawincluding the doctrine of equivalents.

The invention claimed is:
 1. A vehicle control system for a vehicle thatis operable to control the vehicle in an autonomous or semi-autonomousmode, said vehicle control system comprising: a plurality of sensorsdisposed at the vehicle and having fields of sensing exterior of thevehicle; a processor operable to process data captured by said sensors;wherein, when said vehicle control system is operating in the autonomousor semi-autonomous mode and responsive to a determination of atriggering event that requires the vehicle control system to hand overcontrol of the vehicle to a driver of the vehicle before the vehicleencounters an event point, said vehicle control system determines (i) atotal action time until the vehicle encounters the event point, (ii) anestimated time for the driver to take over control of the vehicle and(iii) an estimated handling time for the vehicle to be controlled beforethe vehicle encounters the event point; and wherein, responsive to thedetermined total action time, estimated driver take over time andestimated handling time, said vehicle control system one of (i) allowsthe driver to take over control of the vehicle and (ii) controls thevehicle to slow down and stop the vehicle before the event point.
 2. Thevehicle control system of claim 1, wherein, responsive to the estimateddriver takeover time being greater than the difference between the totalaction time and the estimated handling time, said vehicle control systemcontrols the vehicle to slow down and stop the vehicle.
 3. The vehiclecontrol system of claim 2, wherein a location where the vehicle isslowing down and stopping at is an emergency stopping lane or a shoulderof a road the vehicle is travelling along.
 4. The vehicle control systemof claim 2, wherein a location where the vehicle is slowing down andstopping at is a parking bay or parking lot.
 5. The vehicle controlsystem of claim 1, wherein, responsive to the estimated driver takeovertime being less than the difference between the total action time andthe estimated handling time, said vehicle control system allows thedriver to take over control of the vehicle.
 6. The vehicle controlsystem of claim 5, wherein, responsive to the driver not taking overcontrol of the vehicle by the difference between the total action timeand the estimated handling time, said vehicle control system controlsthe vehicle to slow down and stop the vehicle.
 7. The vehicle controlsystem of claim 6, wherein a location where the vehicle is slowing downand stopping at is an emergency stopping lane or a shoulder of a roadthe vehicle is travelling along.
 8. The vehicle control system of claim6, wherein a location where the vehicle is slowing down and stopping atis a parking bay or parking lot.
 9. The vehicle control system of claim1, wherein the event point comprises an event that cannot be handled bysaid vehicle control system when said vehicle control system isoperating in the autonomous or semi-autonomous mode.
 10. The vehiclecontrol system of claim 1, wherein the triggering event comprisesdetection of an object ahead of the vehicle and in a current path oftravel of the vehicle, and wherein the event point comprises thedetected object.
 11. The vehicle control system of claim 1, wherein thetriggering event is a determination of a hazardous driving conditionahead of the vehicle, and wherein the event point is a location of thedetermined hazardous driving condition.
 12. The vehicle control systemof claim 1, wherein the triggering event is a determination of an errorin said vehicle control system that may affect said control system'sability to control the vehicle in the autonomous or semi-autonomousmode, and the event point is a time or location by which the vehicleshould be stopped.
 13. A vehicle control system for a vehicle that isoperable to control the vehicle in an autonomous or semi-autonomousmode, said vehicle control system comprising: a plurality of sensorsdisposed at the vehicle and having fields of sensing exterior of thevehicle, wherein said plurality of sensors comprises a plurality ofcameras disposed at the vehicle and a plurality of radar sensorsdisposed at the vehicle; a processor operable to process data capturedby said sensors; wherein said vehicle control system determines anupcoming event that cannot be handled by said vehicle control systemwhen said vehicle control system is operating in the autonomous orsemi-autonomous mode; wherein, when said vehicle control system isoperating in the autonomous or semi-autonomous mode and responsive to adetermination of the upcoming event that requires the vehicle controlsystem to hand over control of the vehicle to a driver of the vehiclebefore the vehicle encounters the event, said control determines (i) atotal action time until the vehicle encounters the event, (ii) anestimated time for the driver to take over control of the vehicle and(iii) an estimated handling time for the vehicle to be controlled beforethe vehicle encounters the event; and wherein, responsive to thedetermined total action time, estimated driver take over time andestimated handling time, said vehicle control system one of (i) allowsthe driver to take over control of the vehicle and (ii) controls thevehicle to slow down and stop the vehicle before the vehicle encountersthe event.
 14. The vehicle control system of claim 13, wherein,responsive to the estimated driver takeover time being greater than thedifference between the total action time and the estimated handlingtime, said vehicle control system controls the vehicle to slow down andstop the vehicle.
 15. The vehicle control system of claim 13, wherein,responsive to the estimated driver takeover time being less than thedifference between the total action time and the estimated handlingtime, said vehicle control system allows the driver to take over controlof the vehicle.
 16. The vehicle control system of claim 15, wherein,responsive to the driver not taking over control of the vehicle by thedifference between the total action time and the estimated handlingtime, said vehicle control system controls the vehicle to slow down andstop the vehicle.
 17. The vehicle control system of claim 13, whereinthe event comprises a hazardous driving condition ahead of the vehicle.18. The vehicle control system of claim 13, wherein the event comprisesan imminent failure in said vehicle control system that may affect saidcontrol system's ability to control the vehicle in the autonomous orsemi-autonomous mode.
 19. A vehicle control system for a vehicle that isoperable to control the vehicle in an autonomous or semi-autonomousmode, said vehicle control system comprising: a plurality of sensorsdisposed at the vehicle and having fields of sensing exterior of thevehicle, wherein said plurality of sensors comprises a plurality ofcameras disposed at the vehicle and a plurality of radar sensorsdisposed at the vehicle; a processor operable to process data capturedby said sensors; wherein said vehicle control system determines anupcoming event that cannot be handled by said vehicle control systemwhen said vehicle control system is operating in the autonomous orsemi-autonomous mode; wherein, when said vehicle control system isoperating in the autonomous or semi-autonomous mode and responsive to adetermination of the upcoming event that requires the vehicle controlsystem to hand over control of the vehicle to a driver of the vehiclebefore the vehicle encounters the event, said control determines (i) atotal action time until the vehicle encounters the event, (ii) anestimated time for the driver to take over control of the vehicle and(iii) an estimated handling time for the vehicle to be controlled beforethe vehicle encounters the event; wherein, responsive to the determinedtotal action time, estimated driver take over time and estimatedhandling time, said vehicle control system one of (i) allows the driverto take over control of the vehicle and (ii) controls the vehicle toslow down and stop the vehicle before the vehicle encounters the event;wherein, responsive to the estimated driver takeover time being greaterthan the difference between the total action time and the estimatedhandling time, said vehicle control system controls the vehicle to slowdown and stop the vehicle; wherein, responsive to the estimated drivertakeover time being less than the difference between the total actiontime and the estimated handling time, said vehicle control system allowsthe driver to take over control of the vehicle; and wherein, responsiveto said control allowing the driver to take over control of the vehicleand responsive to the driver not taking over control of the vehicle bythe difference between the total action time and the estimated handlingtime, said vehicle control system controls the vehicle to slow down andstop the vehicle.
 20. The vehicle control system of claim 19, whereinthe event comprises one of (i) a hazardous driving condition ahead ofthe vehicle and (ii) an imminent failure in said vehicle control systemthat may affect said control system's ability to control the vehicle inthe autonomous or semi-autonomous mode.